Incremental Learning Using a Grow-and-Prune Paradigm With Efficient Neural Networks

Deep neural networks (DNNs) have become a widely deployed model for numerous machine learning applications. However, their fixed architecture, substantial training cost, and significant redundancy make it difficult to efficiently update them accommodate previously unseen data. To solve these problems, we propose an incremental framework based on grow-and-prune network synthesis paradigm. When new data arrive, the first grows connections gradients increase capacity Then, iteratively prunes away magnitude of weights enhance compactness, hence recover efficiency. Finally, rests at lightweight DNN that is both ready inference suitable future grow- and-prune updates. The proposed improves accuracy, shrinks size, significantly reduces additional cost incoming compared conventional approaches, such as from scratch fine-tuning. For LeNet-300-100 (LeNet-5) architectures derived MNIST dataset, by up 64 (67), 63 (63), 69 (73 percent) scratch, fine-tuning, respectively. ResNet-18 architecture ImageNet dataset (DeepSpeech2 AN4 dataset), corresponding reductions against fine-tunning, are 60 (62), 72 (71 percent), Our models contain fewer parameters but achieve higher accuracy relative baselines.